A presentation for Research Data at the Edge, Day One of Duke Research Computing Symposium
Why?
- The Web has lots of stuff
- frontier beyond curated datasets
- stuff is wrapped in HTML
- HTML is transported over HTTP but composed for h2m consumption
- Intellectual Property rights bear serious consideration
API
Application Program Interface
- Built for machine-to-machine interactions
- Instructions for programs
Client / Server
- Make [R] interface with the web
- Same as h2m but now m2m
Human Simulation
A dramatization…
JSON
- Javascript Object Notation is a language-independent data format
- Currently the most common data data format for asynchronous client/server communication format
- Consists of key-value pairs
# from https://en.wikipedia.org/wiki/JSON
{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 25,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "office",
"number": "646 555-4567"
},
{
"type": "mobile",
"number": "123 456-7890"
}
],
"children": [],
"spouse": null
}
LS0tDQp0aXRsZTogIlVzaW5nIFIgdG8gT3JjaGVzdHJhdGUgQVBJcyINCmF1dGhvcjogIkpvaG4gTGl0dGxlIg0KZGF0ZTogJ2ByIFN5cy5EYXRlKClgJw0Kb3V0cHV0Og0KICBzbGlkeV9wcmVzZW50YXRpb246IGRlZmF1bHQNCiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdA0KLS0tDQpBIHByZXNlbnRhdGlvbiBmb3IgW1Jlc2VhcmNoIERhdGEgYXQgdGhlIEVkZ2VdKGh0dHA6Ly9saWJyYXJ5LmR1a2UuZWR1L2VkZ2UvZXZlbnRzL3JjMTcpLCBEYXkgT25lIG9mIFtEdWtlIFJlc2VhcmNoIENvbXB1dGluZyBTeW1wb3NpdW1dKGh0dHBzOi8vcmMuZHVrZS5lZHUvc3ltcG9zaXVtLTIwMTcvKQ0KDQoqIFJlcG8NCiogU2xpZGVzDQoqIE5vdGVib29rDQoNCiMjIE91dGxpbmUNCg0KKiBBUEkNCiogSlNPTg0KKiBSIC8gUlN0dWRpbw0KDQojIyBXaHk/DQoNCiogVGhlIFdlYiBoYXMgbG90cyBvZiBzdHVmZg0KICAgICsgZnJvbnRpZXIgYmV5b25kIGN1cmF0ZWQgZGF0YXNldHMNCiAgICArIHN0dWZmIGlzIHdyYXBwZWQgaW4gSFRNTA0KICAgICsgSFRNTCBpcyB0cmFuc3BvcnRlZCBvdmVyIEhUVFAgYnV0IGNvbXBvc2VkIGZvciBoMm0gY29uc3VtcHRpb24NCiAgICArIEludGVsbGVjdHVhbCBQcm9wZXJ0eSByaWdodHMgYmVhciBzZXJpb3VzIGNvbnNpZGVyYXRpb24NCg0KPCEtLSBOQVNBIGFuaW1hdGVkIEdJRiAvLy8gIGh0dHA6Ly9pLmdpcGh5LmNvbS9sMkpodDRsSWZFUWZKM3pqMi5naWYgICAgLS0+IA0KPCEtLSAgZ29vZCBodW1hbiBoYW5kc2hha2UgLy8vICBodHRwOi8vZ2lwaHkuY29tL2dpZnMvdGhvbWFzLVUyWGJvUnVOODlJZGkgLS0+DQo8IS0tIGFmdGVyIHRoZSByZXNlYXJjaCBoYW5kc2hha2UgaXMgY29tcGxldGUgLy8vIGh0dHA6Ly9naXBoeS5jb20vZ2lmcy84MHMtMTk4MHMtdGhvbWFzLWRvbGJ5LXdDS21CZDdvTnRBNGcgIC0tPiANCjwhLS0gdGhlIGNvbmZ1c2lvbiBvZiB0aGUgbTJtIGhhbmRzaGFrZSAvLy8gICBodHRwOi8vZ2lwaHkuY29tL2dpZnMvdGhvbWFzLU1qa0NZak00Nk5yck8gLS0+DQoNCg0KIyMgQVBJDQpBcHBsaWNhdGlvbiBQcm9ncmFtIEludGVyZmFjZSANCg0KKiBCdWlsdCBmb3IgbWFjaGluZS10by1tYWNoaW5lIGludGVyYWN0aW9ucw0KKiBJbnN0cnVjdGlvbnMgZm9yIHByb2dyYW1zDQoNCjwhLS0gaHR0cDovL21vYmlsZS1ncHMubmV0LzIwMTUvMDEvIC0tPg0KIVtdKGltYWdlcy9hcGkucG5nKQ0KDQoNCi0tLSAgICANCg0KIyMjIENsaWVudCAvIFNlcnZlciANCg0KDQohW10oaW1hZ2VzL0NsaWVudC1zZXJ2ZXItbW9kZWwuc3ZnLnBuZykgDQoNCiogTWFrZSBbUl0gaW50ZXJmYWNlIHdpdGggdGhlIHdlYg0KKiBTYW1lIGFzIGgybSBidXQgbm93IG0ybQ0KDQoNCjwhLS0gaHR0cHM6Ly9waXhhYmF5LmNvbS9lbi9jbGllbnQtc2VydmVyLW5ldHdvcmtpbmctbGFwdG9wLTM0MTQyMC8gLS0+DQotLS0gIA0KDQojIyMgSHVtYW4gU2ltdWxhdGlvbg0KDQpBIGRyYW1hdGl6YXRpb24uLi4NCg0KKiBQZXJzb24gdXNlcyBXZWIgQ2xpZW50DQogICAgKyBQZXJzb24gZW50ZXJzIGEgVVJMPGJyPg0KICAgICFbXShpbWFnZXMvVVJMLlBORykNCiAgICANCiAgICArIGNsaWVudCAmIHNlcnZlciBuZWdvdGlhdGU8YnI+IA0KICAgICFbZHJhbWF0aXphdGlvbjogZ29vZCBoYW5kc2hha2VdKGltYWdlcy9nb29kLWhhbmRzaGFrZS5naWYpIA0KICAgICsgSW5mb3JtYXRpb24gaXMgc2VudCBiYWNrIGluIHdyYXBwZWQgSFRNTA0KICAgICsgV2ViIEJyb3dzZXIgcGFyc2VzIHRoZSBIVE1MIA0KICAgIA0KPCEtLSBodHRwczovL2NvbW1vbnMud2lraW1lZGlhLm9yZy93aWtpL0ZpbGU6VW5pZm9ybV9SZXNvdXJjZV9Mb2NhdG9yXyhVUkwpX2V4YW1wbGUuUE5HIC0tPg0KPCEtLSBodHRwczovL2NvbW1vbnMud2lraW1lZGlhLm9yZy93aWtpL0ZpbGU6SFRNTC5zdmcgLS0+DQoNCiMjIG0ybSAtLSBkZXZlbG9wbWVudA0KDQoNCiFbZHJhbWF0aXphdGlvbjogY29uZnVzZWQgYWJvdXQgdGhlIHByb3RvY29sXShpbWFnZXMvZGV2ZWxvcG1lbnQtY29uZnVzaW9uLmdpZikNCiAgICANCiMjIEpTT04NCg0KKiBbSmF2YXNjcmlwdCBPYmplY3QgTm90YXRpb25dKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0pTT04pIGlzIGEgbGFuZ3VhZ2UtaW5kZXBlbmRlbnQgZGF0YSBmb3JtYXQNCiogQ3VycmVudGx5IHRoZSBtb3N0IGNvbW1vbiBkYXRhIGRhdGEgZm9ybWF0IGZvciBhc3luY2hyb25vdXMgY2xpZW50L3NlcnZlciBjb21tdW5pY2F0aW9uIGZvcm1hdA0KKiBDb25zaXN0cyBvZiBrZXktdmFsdWUgcGFpcnMNCg0KPCEtLSBodHRwOi8vaS52aW1lb2Nkbi5jb20vdmlkZW8vNTQxOTM1ODE2XzEyODB4NzIwLmpwZyAtLT4NCjwhLS0gVmltZW8gb24gV2hhdCBpcyBKU09OIC8vIGh0dHBzOi8vdmltZW8uY29tLzE0NDE2MjEwMiAtLT4NCg0KDQpgYGB7anNvbiBleGFtcGxlfQ0KIyBmcm9tIGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0pTT04NCnsNCiAgImZpcnN0TmFtZSI6ICJKb2huIiwNCiAgImxhc3ROYW1lIjogIlNtaXRoIiwNCiAgImlzQWxpdmUiOiB0cnVlLA0KICAiYWdlIjogMjUsDQogICJhZGRyZXNzIjogew0KICAgICJzdHJlZXRBZGRyZXNzIjogIjIxIDJuZCBTdHJlZXQiLA0KICAgICJjaXR5IjogIk5ldyBZb3JrIiwNCiAgICAic3RhdGUiOiAiTlkiLA0KICAgICJwb3N0YWxDb2RlIjogIjEwMDIxLTMxMDAiDQogIH0sDQogICJwaG9uZU51bWJlcnMiOiBbDQogICAgew0KICAgICAgInR5cGUiOiAiaG9tZSIsDQogICAgICAibnVtYmVyIjogIjIxMiA1NTUtMTIzNCINCiAgICB9LA0KICAgIHsNCiAgICAgICJ0eXBlIjogIm9mZmljZSIsDQogICAgICAibnVtYmVyIjogIjY0NiA1NTUtNDU2NyINCiAgICB9LA0KICAgIHsNCiAgICAgICJ0eXBlIjogIm1vYmlsZSIsDQogICAgICAibnVtYmVyIjogIjEyMyA0NTYtNzg5MCINCiAgICB9DQogIF0sDQogICJjaGlsZHJlbiI6IFtdLA0KICAic3BvdXNlIjogbnVsbA0KfQ0KYGBgDQoNCg0KIyMgRXhhbXBsZQ0KDQojIyMgT01EQiBhcGkgDQpsaWtlIGh0dHA6Ly9pbWRiLmNvbS8NCg0KIyMgUiBQYWNrYWdlcyAtLSBSZWxhdGVkDQoNCipQZW9wbGUgd2hvIHVzZSBKU09ObGl0ZSBhbHNvIHVzZS4uLioNCg0KKiBodHRyIC0tIGNhbGxzIEpTT05saXRlIGluIHNlcnZpY2UgdG8gbWFqb3IgZ29hbCBvZiBtYW5hZ2luZyBIVFRQIA0KKiBydmVzdCAtLSAgdXNlZCBmb3IgaHRtbCBwYXJzaW5nDQoNCg0KDQo=